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We conducted this conceptual study to determine if the Institute of Education 
Sciences/National Science Foundation pipeline of evidence guidelines could be applied as a 
protocol that researchers could follow in establishing evidence of effective instructional 
practices. To do this, we compared these guidelines, new drug development process, and our 
own research on major methodological designs and found that they show remarkable 
consistency in the process by which types of studies intended to answer different research 
questions build a body of evidence for practice, whether that practice is in the instructional 
environment or health care environment. However, none of the protocols offers a 
constellation of studies at each stage that would be essential for movement to the next stage 
or the indicators of quality for each type of study. The goal of this effort is to develop 
consensus in the educational research community about a pipeline of evidence protocol that 
provides educators with confidence that the instructional practices they employ have a high 
likelihood of success and will enable a positive impact on the student’s learning and, in the 
broader context, the student’s ability to contribute to society. 
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Introduction 

In the United States, the national debate on the meaning and value of scientific research has 
involved a spectrum of views ranging from those of public policymakers who pass legislation that is 
not informed by educational research to scholars who describe educational research as a complex set 
of methodologies necessary to match to complex research questions. No Child Left Behind 
(reauthorization of the Elementary and Secondary Education Act of 2001 and the more recent Every 
Student Succeeds Act of 2015); the passage of the Education Sciences Reform Act of 2002, which 
established the Institute of Education Sciences (IES) within the U.S. Department of Education; and 
subsequent legislation have been catalysts in this debate. 

Given concerns about the quality of research in education and questions about what constitutes 
evidence for informing instructional practice, several preeminent professional organizations 
developed standards for quality research including the American Educational Research Association 
(2006), American Association of Colleges for Teacher Education (2006), Council for Exceptional 
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Children (2005, 2014), and Division 16 of the American Psychological Association and the Society for 
the Study of School Psychology (2003). In spite of considerable discussion in the literature, at 
conferences and meetings, and in online venues about evidence-based practices, no consensus has 
emerged about what constitutes sufficient evidence to identify an educational practice as research- 
based, with proposed algorithms involving dissimilar configurations of quantities, qualities, and 
types of research (e.g., Cooper, 2010; Gersten et al., 2005; Kazdin, 2011; What Works Clearinghouse, 
2010 ). 

In August of 2013, the IES and the National Science Foundation (NSF) issued common guidelines for 
education research and development. Their purpose was “to identify the spectrum of study types that 
contribute to development and testing of interventions and strategies, and to specify expectations for 
the contributions of each type of study” (IES & NSF, 2013, p. 8). The report describes relevant 
educational research as forming a pipeline of evidence that contributes to the accumulation of 
empirical evidence and development of theoretical models. This is the newest entry in the debate 
about evidence-based practices in education, and given that these federal agencies distribute 
millions of dollars in grants to support research that meets their defined criteria of quality research 
and promising evidence for practice, their guidelines will likely be highly influential among 
educational researchers. 

It is timely for those engaged in educational research to determine if there is a common denominator 
protocol by which the use of particular methodological designs in a line of research inquiry provides 
evidence for each successive step in the process of bringing any given instructional intervention into 
practice. If schools are agents of social change, as Dewey (1937) proposed almost a century ago, then 
it is crucial for students to be immersed in educational practices that enable them to be 
knowledgeable about the global society in which they live, think critically about their environment, 
communicate effectively, and feel empowered to take action that will affect social change. We suggest 
that effective instructional practices are essential for achieving these learning outcomes. 

It is clear that identification of evidence-based practices should not involve the identical processes 
that researchers and educators have historically employed to identify “best practices.” Best practices 
emerged from a personal perspective on the value of a particular instructional practice based on 
experience, predisposition, and interpretations of a particular body of literature. There has never 
been consensus around criteria for what constitutes evidence for best practices. Alternatively, 
evidence-based practices are intended to emerge from verifiable, scientific evidence for effectiveness. 
The research community may argue about what kinds of research are scientific but the intent is to 
seek convergence on criteria rather than to continue relying on disparate notions on the nature of 
scientific evidence. Establishing practices as evidence based should not only provide teachers and 
other stakeholders with a more objective and complete indication of which practices can be 
considered evidence based, but may also begin to change perceptions regarding the trustworthiness 
and relevance of educational research. 

The purpose of this conceptual piece is to determine whether the recent report issued jointly by the 
IES and NSF provides a reasonable protocol that explains the differential role of methodological 
designs in lines of research inquiry that culminate in evidence-based instructional practices. 
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Mode of Inquiry and Data Sources 

We began by identifying three premises that would serve to guide the establishment of a protocol. 

1. We took the position that no one methodological design is superior to any other design. Each 
serves a different purpose within lines of research that lead to greater understanding of 
learning and for identifying practices that are increasingly more effective in teaching. 
Although some researchers might argue that the primary purpose of research is to pursue 
truth rather than to identify evidence-based practices, most would agree that educational 
practices should be informed by research. So the question is how to use research to inform 
educational practice. 

2. The term line of research inquiry connotes building a body of knowledge from study to study. 
Researchers always begin a new investigation by reviewing the prior research on the topic to 
situate a new study within a context of what is already known to move knowledge about the 
topic forward. The term also implies that knowledge derived from research proceeds from 
observing and describing phenomena, to uncovering the links between phenomena, to 
influencing phenomena to generate particular outcomes. 

3. The principle of converging evidence has been proposed as a means for drawing on the 
findings from studies employing different designs to conclude whether a practice is research- 
based. The import given to studies is ultimately in the hands of the professional, though 
many would put greatest weight on experimental studies in any mix of converging evidence 
as these designs are the only ones that can offer claims of causality. As Stanovich and 
Stanovich (2003) noted, 

Issues are most often decided when the community of scientists gradually begins to agree 
that the preponderance of evidence supports one alternative theory rather than another 
... The idea of converging evidence tells us to examine the pattern of flaws running 
through the research literature because the nature of this pattern can either support or 
undermine the conclusions that we might draw. (p. 16) 

We then reviewed four potential protocols for identifying evidence-based practices. One is the IES 
and NSF’s pipeline of evidence guidelines (IES & NSF; 2013); the second is the new drug 
development process established by the U.S. Food and Drug Administration (2015), which is 
considered to be the gold standard of research protocols; the third is the National Institutes of 
Health’s (2007) guidelines for the conduct of research; and the fourth emerged from our research into 
the major methodological designs used in literacy and special education research from which we 
developed a logical and iterative connection between research designs (Schirmer, Lockman, & 
Schirmer, 2013, 2015; Schirmer, Schirmer, & Lockman, 2008). We used thelES/NSF pipeline of 
evidence guidelines as a template on which to overlay the other three to determine if one protocol 
emerges. Table 1 provides a summary of the comparison between each of these. 
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Table 1: Comparison of Line of Research Protocols 

Major Methodological Designs New Drug Development and NIH Guidelines for 
IES/NSF Pipeline of Line of Research Protocol Review Process (U.S. Food the Conduct of 

Evidence _ (Authors) _ and Drug Administration) _ Research _ 

Foundational research: to Exploratory (qualitative Discovery and development 

develop/refine theory and studies) and descriptive of new drugs 

examine phenomena research designs (survey, 

comparison, synthetic 
literature reviews, meta¬ 
analyses, and qualitative 
studies) to identify and 
describe key variables 

Early stage or exploratory Predictive (correlational and Preclinical research: testing 

research: to examine the causal-comparative studies) to on nonhumans to determine 

connections or relationships determine the relationship the likelihood of safety and 

among constructs between variables effectiveness 

Design and development Explanatory (group Clinical research Phase 1: 

research: to draw on theory experimental and single testing for several months 

and evidence from subject experimental studies) on a small group (up to 100) 

exploratory research to to determine if one variable of healthy volunteers or 

design and test individual (independent variable) will individuals with the disease 

components of an cause a change in another or condition to determine 

intervention variable (dependent variable) safety and dosage 

Efficacy research: to test an Explanatory (group Clinical research Phase 2: 

intervention under ideal experimental and single testing for up to 2 years on a 

circumstances subject experimental studies) larger group (up to several 

to determine if a strategy is hundred) of individuals with 
effective when implemented in the disease or condition to 
a highly controlled or clinical determine efficacy and side 

setting effects 

Explanatory (group Clinical research Phase 3: 

experimental and single testing for up to 4 years on a 

subject experimental studies) large number (up to 3,000) 

to determine if a strategy is of individuals with the with 
effective when implemented in the disease or condition to 
a realistic setting, such as an determine efficacy and 
actual classroom monitoring of adverse 

reactions 

Explanatory (group Clinical research Phase 4: 

experimental studies) to testing on several thousand 

determine if a strategy is individuals with the disease 

effective when implemented in or condition to determine 
a wide range of realistic safety and efficacy, 

settings, such as several 

schools or districts. _ 

Note. IES = Institute of Education Sciences; NSF = National Science Foundation; NIH = National 
Institutes of Health. 


Scale-up research: to test an 
intervention under typical 
circumstances in a wide 
range of contexts and 
populations 


Effectiveness research: to 
test an intervention under 
typical circumstances 


Focus is on the 
responsible use of 
scientific 

methodologies and 
does not address 
particular 
m ethodologic al 
designs 
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Warrants for Point of View 

IES/NSF includes six types of research that form their pipeline of evidence. 

• Research Type 1 is foundational research. These studies are described by IES/NSF as 
providing foundational knowledge of teaching and learning, developing, and refining 
theory and examining phenomena in the absence of a direct link to educational outcomes. 

• Research Type 2 is early stage or exploratory research. These studies examine the 
connections or relationships among constructs that may result in the development of a 
new intervention. 

• Research Type 3 is design and development research. These studies draw on theory and 
empirical evidence in designing an intervention and testing individual components. 

• Research Type 4 is efficacy research. These studies test the intervention under ideal 
circumstances. 

• Research Type 5 is effectiveness research. These studies test the intervention under 
typical circumstances. 

• Research Type 6 is scale-up research. These studies test the intervention under typical 
circumstances but in a wide range of contexts and populations. 

Foundational Research 

The IES/N SF foundational research step is consistent with the exploratory research and descriptive 
research designs that we found are commonly used to categorize two of the major types of 
educational research in the methodology literature. 

Exploratory research is designed to develop insights, ideas, and understandings about a 
phenomenon. The phenomena are not well understood, variables need to be identified or discovered, 
or the purpose is to generate hypotheses for further investigation. Exploratory methodological 
designs are typically qualitative studies. 

Descriptive research is designed to describe the characteristics of a phenomenon, differentiate a 
phenomenon from other phenomena, describe the variables, and document features of the 
phenomenon such as behaviors, events, process, attitudes, and beliefs. Descriptive methodological 
designs are typically survey studies, comparison studies, synthetic literature reviews, meta-analyses, 
and qualitative studies. 

The IES/N SF foundational research step is also consonant with the discovery and development stage 
of the new drug development process, which is the stage in which researchers seek new insights into 
a disease or condition, develop new technologies, identify unanticipated effects of already approved 
treatments, and test promising molecular compounds. 

Early Stage or Exploratory Research 

The IES/N SF early stage or exploratory research step is consonant with what we found is typically 
referred to as Predictive designs in educational research. Predictive research is designed to identify 
the relationship between variables to predict a phenomenon. The variables have previously 
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been identified and described and are now examined in terms of their relationship to one another to 
speculate about one variable influencing the occurrence of others. Predictive methodological designs 
are typically correlational studies and causal-comparative studies. 

It is at this IES/NSF early stage or exploratory research stage that there is a parallel with the 
preclinical research stage of the new drug development process. This stage involves testing the drug 
in vivo, outside a living organism, and in vitro (inside a nonhuman living organism) to establish 
toxicity and dosing levels so that it can be determined whether the drug could cause serious harm. 

Design and Development Research 

The IES/NSF design and development research step is compatible with what we found are 
explanatory designs in educational research. Explanatory research is designed to identify the 
existence of a cause—effect relationship between variables. A previously found relationship between 
variables indicates the likelihood that one causes the other and investigating the likelihood confirms 
or disconfirms the direction and magnitude of the relationship. Explanatory methodological designs 
fall into two categories—experimental group designs and experimental single subject designs. 
Experimental group designs are typically random control trial, quasiexperimental, and single-group 
studies. Experimental single subject designs are typically simple time series, reversal, multiple 
baseline, changing criterion, and multiple treatment studies. 

The clinical research stage of the new drug development process incorporates four phases that are 
quite similar to the IES/N SF protocol and the purpose of the first phase is similar to the IES/N SF 
design and development step. That is, in Phase 1, the drug is tested on a relatively small number of 
20 to 100 healthy volunteers or individuals with the disease or condition over a period of several 
months for the purpose of determining safety and dosage. 

Efficacy Research 

Group and single-subject experimental research designs are also employed for what IES/NSF refers 
to as efficacy research. These studies are designed to determine if an instructional strategy is 
effective when it is implemented in a highly controlled or clinical setting by individuals who are 
trained to carry out the strategy and monitored to ensure they are implementing it with fidelity. 

Phase 2 of the new drug development process corresponds to this efficacy research step. Studies in 
Phase 2 are conducted with up to several hundred individuals who have the disease or condition over 
a period of several months to 2 years for the purpose of determining the efficacy and side effects of 
the drug. 

Effectiveness Research 

Group and single subject experimental research designs are also used for effectiveness research but, 
unlike efficacy research, these studies are designed to determine if an instructional strategy is 
effective when it is implemented in a realistic setting by individuals who would normally be carrying 
out instruction in the setting. Actual classrooms are the typical setting for these studies and the 
instructional intervention is implemented by the teachers. 

The third phase of the new drug development process is comparable to this step of the IES/NSF 
protocol. In this phase, the drug is tested with 300 to 3,000 individuals with the disease or condition 
over a period of 1 to 4 years for the purpose of assessing efficacy and adverse reactions. 
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Scale-Up Research 

Given that scale-up research involves evaluating the effectiveness of an instructional intervention in 
a wide range of typical contexts and populations, scale-up research is almost exclusively the domain 
of group experimental studies that can involve large numbers of participants across a variety of 
schools, districts, and even geographic areas. Scale-up research is quite similar to Phase 4 of the new 
drug development process in which the drug is tested on several thousand individuals with the 
disease or condition for the purpose of assessing safety and efficacy. 

The fourth potential protocol we examined, the NIH guidelines, focuses on the responsible use of 
scientific methodologies in seeking new knowledge and encompasses the responsibilities of research 
supervisors and trainees, acquisition and maintenance of data, role of publication, establishment of 
authorship, function of peer review, importance of collaboration, disclosure of financial conflicts of 
interest, and protection of human subjects. While a critical part of the research process, the NIH 
guidelines do not contribute to understanding how particular designs can be employed to establish a 
line of research protocol that leads to the identification of evidence-based practices. 

Results and Substantiated Conclusions 

We conducted this conceptual study to determine if the IES/NSF pipeline of evidence guidelines 
could be applied as a protocol that researchers could follow in establishing evidence of effective 
instructional practices. To do this, we compared the IES/NSF guidelines, new drug development 
process, and our own research on major methodological designs and found that they show 
remarkable consistency in the process by which types of studies intended to answer different 
research questions build a body of evidence for practice, whether that practice is in the instructional 
environment or health care environment. 

What our examination does not provide is the constellation of studies at each step in the IES/NSF 
pipeline process that warrant movement to the next step and the indicators of quality for each type 
of study to ensure that results are trustworthy and, thus, able to contribute to the pipeline of 
evidence. Without such assurance about trustworthiness of findings, practitioners are unlikely to 
embrace practices with research support any more than approaches with little or no evidence of 
effectiveness that are promoted in the media. 

Clearly, no design is immune to flaws. Indeed, a key reason why research has shown that 
practitioners pay little attention to the published research is based in part on claims for significant 
findings and generalization of results that are often unwarranted given the limitations of the studies 
(Fusarelli, 2008; Goldhaber & Brewer, 2008). Other reasons for the research-to-practice gap include 
limited accessibility to research reports, limited comprehensibility of research articles, limited 
relevance of research to actual practice, and little agreement among researchers about what 
constitutes evidence for claims of effective practice (Springer, 2010; Vaughn, Klingner, & Hughes, 
2000). Thus, it is crucial that in addition to making the case that a practice is research-based 
because evidence has been built from conceptual and descriptive studies to correlational and causal 
studies but also because there are a sufficient number of high quality studies at each step in the 
process. 
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Significance for Social Change 

Examination of thelES/NSF pipeline of evidence guidelines, the new drug development and review 
process established by the U.S. Food and Drug Administration, and the respective role of 
methodological designs found in the educational methodology literature indicate an emerging 
protocol that may lead to consensus about the identification of evidence-based practices. The process 
begins with studies that explore and describe phenomena pertinent to the learner and learning 
environment. As theory is built and key variables are identified, the process moves to studies that 
seek relationships among these variables. Having identified strong relationships, the process can 
move to establishing cause—effect claims through experimental designs. The approaches we 
examined align in recognizing that this is not a linear process but iterative as new questions 
sometimes lead back to previous designs. In this proposed protocol, identification of evidence-based 
practices must culminate in experimental studies that can reveal causality between instructional 
interventions and learning outcomes. However, the process cannot begin with experimental studies, 
for each design serves a crucial role in a line of research inquiry that can ultimately provide 
converging evidence for instructional practices. The next step is to determine the constellation of 
quantity and quality of studies needed to provide evidence that an instructional strategy has 
sufficient evidence of effectiveness. 

Several cautions are warranted when discussing evidence-based practices. Cook and Odom (2013) 
suggested that there are three key issues: 

1. No practice works for every student. Though an instructional strategy 
may work for most students with certain learning characteristics 
taught by teachers who are skilled in delivering the instruction, it 
will undoubtedly not work with every student with the same 
characteristics and teachers. 

2. Insufficient high-quality research examining lines of research on 
instructional practices is available from which evidence-based 
practices can be identified. When the research base in insufficient, it 
is clear that teachers must and do make judgments about which 
practices are promising. The goal is to encourage researchers to 
conduct high-quality studies that build a base of evidence for 
instructional practices. 

3. The research-to-practice gap is in part due to the difficulty of 
implementing and sustaining a new practice in the real world of 
classrooms and schools. No matter how well researched a practice is, 
and how clear findings have been about the characteristics of 
students who benefit and instructional conditions needed to achieve 
the outcomes attained in empirical studies, when applied in the 
messy world of real classrooms, problems and issues will undoubtedly 
arise. Willingness to problem-solve the issues that occur depends on 
the resilience and self-efficacy of those implementing the practice. 
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Certainly, identifying evidence-based practices is not the only reason for carrying out educational 
research. However, it seems undeniable that instruction should be informed by research just as 
health care should be. Thus, the goal is to conduct high quality research that provides educators with 
confidence that the instructional practices they employ have a high likelihood of success and will 
enable a positive impact on the student’s learning and, in the broader context, social change in the 
schools and each student’s ability to contribute to society. 
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